{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Image Captioning with RNNs\n",
    "In this exercise you will implement a vanilla recurrent neural networks and use them it to train a model that can generate novel captions for images.\n",
    "\n",
    "在这个练习中，您将实现一个普通的循环神经网络，并使用它们来训练一个可以为图像生成新标注的模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "# As usual, a bit of setup\n",
    "import time, os, json\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.rnn_layers import *\n",
    "from cs231n.captioning_solver import CaptioningSolver\n",
    "from cs231n.classifiers.rnn import CaptioningRNN\n",
    "from cs231n.coco_utils import load_coco_data, sample_coco_minibatch, decode_captions\n",
    "from cs231n.image_utils import image_from_url\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "    \"\"\" returns relative error \"\"\"\n",
    "    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install h5py\n",
    "The COCO dataset we will be using is stored in HDF5 format. \n",
    "\n",
    "To load HDF5 files, we will need to install the `h5py` Python package. \n",
    "\n",
    "我们将使用的COCO数据集以HDF5格式存储。\n",
    "\n",
    "要加载HDF5文件，我们需要安装`h5py`Python包。\n",
    "\n",
    "From the command line, run: \n",
    "\n",
    "`pip install h5py`  \n",
    "\n",
    "If you receive a permissions error, you may need to run the command as root: \n",
    "\n",
    "如果你收到一个权限错误，你可能需要运行命令作为根:\n",
    "\n",
    "```sudo pip install h5py```\n",
    "\n",
    "You can also run commands directly from the Jupyter notebook by prefixing the command with the \"!\" character:\n",
    "\n",
    "你也可以直接从Jupyter笔记本上运行命令，方法是在命令前面加上\"!\"字符:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install h5py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "source": [
    "# Microsoft COCO\n",
    "For this exercise we will use the 2014 release of the [Microsoft COCO dataset](http://mscoco.org/) which has become the standard testbed for image captioning. \n",
    "\n",
    "The dataset consists of 80,000 training images and 40,000 validation images, each annotated with 5 captions written by workers on Amazon Mechanical Turk.\n",
    "\n",
    "对于这个练习，我们将使用2014年发布的[Microsoft COCO数据集]，它已经成为图像标注的标准测试平台。\n",
    "\n",
    "该数据集包括80,000张训练图像和40,000张验证图像，每个图像都有5个由Amazon Mechanical Turk上的工作人员编写的标注。\n",
    "\n",
    "You should have already downloaded the data by changing to the `cs231n/datasets` directory and running the script `get_assignment3_data.sh`. If you haven't yet done so, run that script now. \n",
    "\n",
    "Warning: the COCO data download is ~1GB.\n",
    "\n",
    "通过切换到`cs231n/datasets`目录并运行脚本`get_assignment3_data.sh`，您应该已经下载了数据。如果您还没有这样做，现在就运行这个脚本。\n",
    "\n",
    "警告:COCO 数据下载是~1GB。\n",
    "\n",
    "We have preprocessed the data and extracted features for you already. \n",
    "\n",
    "For all images we have extracted features from the fc7 layer of the VGG-16 network pretrained on ImageNet; these features are stored in the files `train2014_vgg16_fc7.h5` and `val2014_vgg16_fc7.h5` respectively. \n",
    "\n",
    "To cut down on processing time and memory requirements, we have reduced the dimensionality of the features from 4096 to 512; these features can be found in the files `train2014_vgg16_fc7_pca.h5` and `val2014_vgg16_fc7_pca.h5`.\n",
    "\n",
    "我们已经对数据进行了预处理并为您提取了特征。\n",
    "\n",
    "对于所有的图像，我们提取了在ImageNet上预先训练的VGG-16网络的fc7层的特征;这些特性分别存储在文件`train2014_vgg16_fc7.h5` and `val2014_vgg16_fc7.h5`\n",
    "\n",
    "为了减少处理时间和内存需求，我们将特征的维数从4096降至512;这些特性可以在文件`train2014_vgg16_fc7_pca.h5` and `val2014_vgg16_fc7_pca.h5`中找到\n",
    "\n",
    "The raw images take up a lot of space (nearly 20GB) so we have not included them in the download. \n",
    "\n",
    "However all images are taken from Flickr, and URLs of the training and validation images are stored in the files `train2014_urls.txt` and `val2014_urls.txt` respectively. \n",
    "\n",
    "This allows you to download images on the fly for visualization. \n",
    "\n",
    "Since images are downloaded on-the-fly, **you must be connected to the internet to view images**.\n",
    "\n",
    "原始图像占用了大量空间(近20GB)，所以我们没有在下载中包含它们。\n",
    "\n",
    "但是，所有图像都是从Flickr获取的，训练和验证图像的url存储在文件`train2014_urls.txt` and `val2014_urls.txt`\n",
    "\n",
    "这允许您动态下载图像以实现可视化。\n",
    "\n",
    "由于图像是动态下载的，**您必须连接到互联网才能查看图像**。\n",
    "\n",
    "Dealing with strings is inefficient, so we will work with an encoded version of the captions. \n",
    "\n",
    "Each word is assigned an integer ID, allowing us to represent a caption by a sequence of integers. \n",
    "\n",
    "The mapping between integer IDs and words is in the file `coco2014_vocab.json`, and you can use the function `decode_captions` from the file `cs231n/coco_utils.py` to convert numpy arrays of integer IDs back into strings.\n",
    "\n",
    "处理字符串的效率很低，所以我们将使用字幕的编码版本。\n",
    "\n",
    "每个单词都分配了一个整数ID，允许我们用一个整数序列来表示标注。\n",
    "\n",
    "整数ID和单词之间的映射在文件`coco2014_vocab.json`中。\n",
    "\n",
    "可以使用文件`cs231n/coco_utils.py`中的函数`decode_captions`将整数ID的numpy数组转换回字符串。\n",
    "\n",
    "There are a couple special tokens that we add to the vocabulary. \n",
    "\n",
    "We prepend a special `<START>` token and append an `<END>` token to the beginning and end of each caption respectively. \n",
    "\n",
    "Rare words are replaced with a special `<UNK>` token (for \"unknown\"). \n",
    "\n",
    "In addition, since we want to train with minibatches containing captions of different lengths, we pad short captions with a special `<NULL>` token after the `<END>` token and don't compute loss or gradient for `<NULL>` tokens. \n",
    "\n",
    "Since they are a bit of a pain, we have taken care of all implementation details around special tokens for you.\n",
    "\n",
    "\n",
    "我们向词汇表中添加了一些特殊的标记。\n",
    "\n",
    "我们在每个标题的开头和结尾分别添加一个特殊的`<START>`令牌和`<END>`令牌。\n",
    "\n",
    "罕见的单词被替换为特殊的`<UNK>`令牌(表示“未知”)。\n",
    "\n",
    "此外，由于我们希望使用包含不同长度标题的小批量进行训练，所以我们在`<END>`标记之后用特殊的`<NULL>`标记填充短标题，并且不计算`<NULL>`标记的丢失或梯度。\n",
    "\n",
    "由于它们有点麻烦，我们已经为您处理了关于特殊令牌的所有实现细节。\n",
    "\n",
    "You can load all of the MS-COCO data (captions, features, URLs, and vocabulary) using the `load_coco_data` function from the file `cs231n/coco_utils.py`. \n",
    "\n",
    "可以使用`cs231n/coco_utils.py`文件中的`load_coco_data`函数加载所有MS-COCO数据(标注、特性、URLs和词汇表)。\n",
    "\n",
    "Run the following cell to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "# Load COCO data from disk; this returns a dictionary\n",
    "# We'll work with dimensionality-reduced features for this notebook, but feel\n",
    "# free to experiment with the original features by changing the flag below.\n",
    "data = load_coco_data(pca_features=True)\n",
    "\n",
    "# Print out all the keys and values from the data dictionary\n",
    "for k, v in data.items():\n",
    "    if type(v) == np.ndarray:\n",
    "        print(k, type(v), v.shape, v.dtype)\n",
    "    else:\n",
    "        print(k, type(v), len(v))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Look at the data\n",
    "It is always a good idea to look at examples from the dataset before working with it.\n",
    "\n",
    "You can use the `sample_coco_minibatch` function from the file `cs231n/coco_utils.py` to sample minibatches of data from the data structure returned from `load_coco_data`. Run the following to sample a small minibatch of training data and show the images and their captions. Running it multiple times and looking at the results helps you to get a sense of the dataset.\n",
    "\n",
    "Note that we decode the captions using the `decode_captions` function and that we download the images on-the-fly using their Flickr URL, so **you must be connected to the internet to view images**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample a minibatch and show the images and captions\n",
    "batch_size = 3\n",
    "\n",
    "captions, features, urls = sample_coco_minibatch(data, batch_size=batch_size)\n",
    "for i, (caption, url) in enumerate(zip(captions, urls)):\n",
    "    plt.imshow(image_from_url(url))\n",
    "    plt.axis('off')\n",
    "    caption_str = decode_captions(caption, data['idx_to_word'])\n",
    "    plt.title(caption_str)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recurrent Neural Networks\n",
    "As discussed in lecture, we will use recurrent neural network (RNN) language models for image captioning. \n",
    "\n",
    "The file `cs231n/rnn_layers.py` contains implementations of different layer types that are needed for recurrent neural networks, and the file `cs231n/classifiers/rnn.py` uses these layers to implement an image captioning model.\n",
    "\n",
    "正如在课堂上所讨论的，我们将使用递归神经网络(RNN)语言模型进行图像字幕。\n",
    "\n",
    "文件的`cs231n/rnn_layers.py`包含循环神经网络所需的不同层类型的实现，以及文件`cs231n/classifiers/rnn.py`使用这些层来实现一个图像标注模型。\n",
    "\n",
    "We will first implement different types of RNN layers in `cs231n/rnn_layers.py`.\n",
    "\n",
    "我们将首先实现不同类型的RNN层在`cs231n/rnn_layers.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vanilla RNN: step forward\n",
    "Open the file `cs231n/rnn_layers.py`. This file implements the forward and backward passes for different types of layers that are commonly used in recurrent neural networks.\n",
    "\n",
    "打开文件`cs231n/rnn_layers.py`。该文件实现了循环神经网络中常用的不同类型的层的前向和反向传播。\n",
    "\n",
    "First implement the function `rnn_step_forward` which implements the forward pass for a single timestep of a vanilla recurrent neural network. \n",
    "\n",
    "首先实现函数`rnn_step_forward`，该函数实现普通循环神经网络单个时间步长的前向传播。\n",
    "\n",
    "After doing so run the following to check your implementation. \n",
    "\n",
    "这样做之后，运行以下代码来检查您的实现。\n",
    "\n",
    "You should see errors on the order of e-8 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "N, D, H = 3, 10, 4\n",
    "\n",
    "x = np.linspace(-0.4, 0.7, num=N*D).reshape(N, D)\n",
    "prev_h = np.linspace(-0.2, 0.5, num=N*H).reshape(N, H)\n",
    "Wx = np.linspace(-0.1, 0.9, num=D*H).reshape(D, H)\n",
    "Wh = np.linspace(-0.3, 0.7, num=H*H).reshape(H, H)\n",
    "b = np.linspace(-0.2, 0.4, num=H)\n",
    "\n",
    "next_h, _ = rnn_step_forward(x, prev_h, Wx, Wh, b)\n",
    "expected_next_h = np.asarray([\n",
    "  [-0.58172089, -0.50182032, -0.41232771, -0.31410098],\n",
    "  [ 0.66854692,  0.79562378,  0.87755553,  0.92795967],\n",
    "  [ 0.97934501,  0.99144213,  0.99646691,  0.99854353]])\n",
    "\n",
    "print('next_h error: ', rel_error(expected_next_h, next_h))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vanilla RNN: step backward\n",
    "In the file `cs231n/rnn_layers.py` implement the `rnn_step_backward` function. \n",
    "\n",
    "After doing so run the following to numerically gradient check your implementation. \n",
    "\n",
    "在文件`cs231n/rnn_layers.py`中。实现`rnn_step_backward`函数。\n",
    "\n",
    "这样做之后，运行下面的代码来数值梯度检查你的实现。\n",
    "\n",
    "You should see errors on the order of `e-8` or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cs231n.rnn_layers import rnn_step_forward, rnn_step_backward\n",
    "np.random.seed(231)\n",
    "N, D, H = 4, 5, 6\n",
    "x = np.random.randn(N, D)\n",
    "h = np.random.randn(N, H)\n",
    "Wx = np.random.randn(D, H)\n",
    "Wh = np.random.randn(H, H)\n",
    "b = np.random.randn(H)\n",
    "\n",
    "out, cache = rnn_step_forward(x, h, Wx, Wh, b)\n",
    "\n",
    "dnext_h = np.random.randn(*out.shape)\n",
    "\n",
    "fx = lambda x: rnn_step_forward(x, h, Wx, Wh, b)[0]\n",
    "fh = lambda prev_h: rnn_step_forward(x, h, Wx, Wh, b)[0]\n",
    "fWx = lambda Wx: rnn_step_forward(x, h, Wx, Wh, b)[0]\n",
    "fWh = lambda Wh: rnn_step_forward(x, h, Wx, Wh, b)[0]\n",
    "fb = lambda b: rnn_step_forward(x, h, Wx, Wh, b)[0]\n",
    "\n",
    "dx_num = eval_numerical_gradient_array(fx, x, dnext_h)\n",
    "dprev_h_num = eval_numerical_gradient_array(fh, h, dnext_h)\n",
    "dWx_num = eval_numerical_gradient_array(fWx, Wx, dnext_h)\n",
    "dWh_num = eval_numerical_gradient_array(fWh, Wh, dnext_h)\n",
    "db_num = eval_numerical_gradient_array(fb, b, dnext_h)\n",
    "\n",
    "dx, dprev_h, dWx, dWh, db = rnn_step_backward(dnext_h, cache)\n",
    "\n",
    "print('dx error: ', rel_error(dx_num, dx))\n",
    "print('dprev_h error: ', rel_error(dprev_h_num, dprev_h))\n",
    "print('dWx error: ', rel_error(dWx_num, dWx))\n",
    "print('dWh error: ', rel_error(dWh_num, dWh))\n",
    "print('db error: ', rel_error(db_num, db))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vanilla RNN: forward\n",
    "Now that you have implemented the forward and backward passes for a single timestep of a vanilla RNN, you will combine these pieces to implement a RNN that processes an entire sequence of data.\n",
    "\n",
    "既然您已经实现了普通RNN的单个时间步长的前向和反向传播，那么您将组合这些片段来实现处理整个数据序列的RNN。\n",
    "\n",
    "In the file `cs231n/rnn_layers.py`, implement the function `rnn_forward`. \n",
    "\n",
    "在文件`cs231n/rnn_layers.py`中实现函数`rnn_forward`。\n",
    "\n",
    "This should be implemented using the `rnn_step_forward` function that you defined above. \n",
    "\n",
    "这应该使用上面定义的`rnn_step_forward`函数来实现。\n",
    "\n",
    "After doing so run the following to check your implementation. \n",
    "\n",
    "这样做之后，运行以下代码来检查您的实现。\n",
    "\n",
    "You should see errors on the order of `e-7` or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "N, T, D, H = 2, 3, 4, 5\n",
    "\n",
    "x = np.linspace(-0.1, 0.3, num=N*T*D).reshape(N, T, D)\n",
    "h0 = np.linspace(-0.3, 0.1, num=N*H).reshape(N, H)\n",
    "Wx = np.linspace(-0.2, 0.4, num=D*H).reshape(D, H)\n",
    "Wh = np.linspace(-0.4, 0.1, num=H*H).reshape(H, H)\n",
    "b = np.linspace(-0.7, 0.1, num=H)\n",
    "\n",
    "h, _ = rnn_forward(x, h0, Wx, Wh, b)\n",
    "expected_h = np.asarray([\n",
    "  [\n",
    "    [-0.42070749, -0.27279261, -0.11074945,  0.05740409,  0.22236251],\n",
    "    [-0.39525808, -0.22554661, -0.0409454,   0.14649412,  0.32397316],\n",
    "    [-0.42305111, -0.24223728, -0.04287027,  0.15997045,  0.35014525],\n",
    "  ],\n",
    "  [\n",
    "    [-0.55857474, -0.39065825, -0.19198182,  0.02378408,  0.23735671],\n",
    "    [-0.27150199, -0.07088804,  0.13562939,  0.33099728,  0.50158768],\n",
    "    [-0.51014825, -0.30524429, -0.06755202,  0.17806392,  0.40333043]]])\n",
    "print('h error: ', rel_error(expected_h, h))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vanilla RNN: backward\n",
    "In the file `cs231n/rnn_layers.py`, implement the backward pass for a vanilla RNN in the function `rnn_backward`. \n",
    "\n",
    "在文件`cs231n/rnn_layers.py`中，在函数`rnn_backward`中实现普通RNN的反向传播。\n",
    "\n",
    "This should run back-propagation over the entire sequence, making calls to the `rnn_step_backward` function that you defined earlier. \n",
    "\n",
    "这应该在整个序列上运行反向传播，调用前面定义的`rnn_step_backward`函数。\n",
    "\n",
    "You should see errors on the order of e-6 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(231)\n",
    "\n",
    "N, D, T, H = 2, 3, 10, 5\n",
    "\n",
    "x = np.random.randn(N, T, D)\n",
    "h0 = np.random.randn(N, H)\n",
    "Wx = np.random.randn(D, H)\n",
    "Wh = np.random.randn(H, H)\n",
    "b = np.random.randn(H)\n",
    "\n",
    "out, cache = rnn_forward(x, h0, Wx, Wh, b)\n",
    "\n",
    "dout = np.random.randn(*out.shape)\n",
    "\n",
    "dx, dh0, dWx, dWh, db = rnn_backward(dout, cache)\n",
    "\n",
    "fx = lambda x: rnn_forward(x, h0, Wx, Wh, b)[0]\n",
    "fh0 = lambda h0: rnn_forward(x, h0, Wx, Wh, b)[0]\n",
    "fWx = lambda Wx: rnn_forward(x, h0, Wx, Wh, b)[0]\n",
    "fWh = lambda Wh: rnn_forward(x, h0, Wx, Wh, b)[0]\n",
    "fb = lambda b: rnn_forward(x, h0, Wx, Wh, b)[0]\n",
    "\n",
    "dx_num = eval_numerical_gradient_array(fx, x, dout)\n",
    "dh0_num = eval_numerical_gradient_array(fh0, h0, dout)\n",
    "dWx_num = eval_numerical_gradient_array(fWx, Wx, dout)\n",
    "dWh_num = eval_numerical_gradient_array(fWh, Wh, dout)\n",
    "db_num = eval_numerical_gradient_array(fb, b, dout)\n",
    "\n",
    "print('dx error: ', rel_error(dx_num, dx))\n",
    "print('dh0 error: ', rel_error(dh0_num, dh0))\n",
    "print('dWx error: ', rel_error(dWx_num, dWx))\n",
    "print('dWh error: ', rel_error(dWh_num, dWh))\n",
    "print('db error: ', rel_error(db_num, db))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word embedding: forward\n",
    "In deep learning systems, we commonly represent words using vectors. \n",
    "\n",
    "在深度学习系统中，我们通常使用向量来表示单词。\n",
    "\n",
    "Each word of the vocabulary will be associated with a vector, and these vectors will be learned jointly with the rest of the system.\n",
    "\n",
    "词汇表中的每个单词都将与一个向量相关联，这些向量将与系统的其他部分共同学习。\n",
    "\n",
    "In the file `cs231n/rnn_layers.py`, implement the function `word_embedding_forward` to convert words (represented by integers) into vectors. \n",
    "\n",
    "在文件`cs231n/rnn_layers.py`中实现函数`word_embedding_forward`将单词(用整数表示)转换为向量。\n",
    "\n",
    "Run the following to check your implementation. \n",
    "\n",
    "You should see an error on the order of `e-8` or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "N, T, V, D = 2, 4, 5, 3\n",
    "\n",
    "x = np.asarray([[0, 3, 1, 2], [2, 1, 0, 3]])\n",
    "W = np.linspace(0, 1, num=V*D).reshape(V, D)\n",
    "\n",
    "out, _ = word_embedding_forward(x, W)\n",
    "expected_out = np.asarray([\n",
    " [[ 0.,          0.07142857,  0.14285714],\n",
    "  [ 0.64285714,  0.71428571,  0.78571429],\n",
    "  [ 0.21428571,  0.28571429,  0.35714286],\n",
    "  [ 0.42857143,  0.5,         0.57142857]],\n",
    " [[ 0.42857143,  0.5,         0.57142857],\n",
    "  [ 0.21428571,  0.28571429,  0.35714286],\n",
    "  [ 0.,          0.07142857,  0.14285714],\n",
    "  [ 0.64285714,  0.71428571,  0.78571429]]])\n",
    "\n",
    "print('out error: ', rel_error(expected_out, out))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word embedding: backward\n",
    "Implement the backward pass for the word embedding function in the function `word_embedding_backward`. \n",
    "\n",
    "在函数`word_embedding_backward`中实现单词嵌入函数的反向传播。\n",
    "\n",
    "After doing so run the following to numerically gradient check your implementation. \n",
    "\n",
    "这样做之后，运行下面的代码来数值梯度检查你的实现。\n",
    "\n",
    "You should see an error on the order of `e-11` or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(231)\n",
    "\n",
    "N, T, V, D = 50, 3, 5, 6\n",
    "x = np.random.randint(V, size=(N, T))\n",
    "W = np.random.randn(V, D)\n",
    "\n",
    "out, cache = word_embedding_forward(x, W)\n",
    "dout = np.random.randn(*out.shape)\n",
    "dW = word_embedding_backward(dout, cache)\n",
    "\n",
    "f = lambda W: word_embedding_forward(x, W)[0]\n",
    "dW_num = eval_numerical_gradient_array(f, W, dout)\n",
    "\n",
    "print('dW error: ', rel_error(dW, dW_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Temporal Affine layer\n",
    "At every timestep we use an affine function to transform the RNN hidden vector at that timestep into scores for each word in the vocabulary. \n",
    "\n",
    "在每一个时间步，我们使用affine函数将RNN隐藏向量转换成词汇表中每个单词的得分。\n",
    "\n",
    "Because this is very similar to the affine layer that you implemented in assignment 2, we have provided this function for you in the `temporal_affine_forward` and `temporal_affine_backward` functions in the file `cs231n/rnn_layers.py`. \n",
    "\n",
    "因为这与您在作业2中实现的affine层非常相似，所以我们在文件`cs231n/rnn_layers.py`中的`temporal_affine_forward`和`temporal_affine_backward`函数中为您提供了这个函数。\n",
    "\n",
    "Run the following to perform numeric gradient checking on the implementation. \n",
    "\n",
    "You should see errors on the order of e-9 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "np.random.seed(231)\n",
    "\n",
    "# Gradient check for temporal affine layer\n",
    "N, T, D, M = 2, 3, 4, 5\n",
    "x = np.random.randn(N, T, D)\n",
    "w = np.random.randn(D, M)\n",
    "b = np.random.randn(M)\n",
    "\n",
    "out, cache = temporal_affine_forward(x, w, b)\n",
    "\n",
    "dout = np.random.randn(*out.shape)\n",
    "\n",
    "fx = lambda x: temporal_affine_forward(x, w, b)[0]\n",
    "fw = lambda w: temporal_affine_forward(x, w, b)[0]\n",
    "fb = lambda b: temporal_affine_forward(x, w, b)[0]\n",
    "\n",
    "dx_num = eval_numerical_gradient_array(fx, x, dout)\n",
    "dw_num = eval_numerical_gradient_array(fw, w, dout)\n",
    "db_num = eval_numerical_gradient_array(fb, b, dout)\n",
    "\n",
    "dx, dw, db = temporal_affine_backward(dout, cache)\n",
    "\n",
    "print('dx error: ', rel_error(dx_num, dx))\n",
    "print('dw error: ', rel_error(dw_num, dw))\n",
    "print('db error: ', rel_error(db_num, db))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Temporal Softmax loss\n",
    "In an RNN language model, at every timestep we produce a score for each word in the vocabulary. \n",
    "\n",
    "We know the ground-truth word at each timestep, so we use a softmax loss function to compute loss and gradient at each timestep. \n",
    "\n",
    "We sum the losses over time and average them over the minibatch.\n",
    "\n",
    "在RNN语言模型中，每一步我们都会为词汇表中的每个单词生成一个分数。\n",
    "\n",
    "我们知道每一时间步的真实字，所以我们使用一个softmax损失函数来计算每一步的损失和梯度。\n",
    "\n",
    "我们把一段时间内的损失加起来，然后把它们平均到小批量上。\n",
    "\n",
    "However there is one wrinkle: since we operate over minibatches and different captions may have different lengths, we append `<NULL>` tokens to the end of each caption so they all have the same length. \n",
    "\n",
    "We don't want these `<NULL>` tokens to count toward the loss or gradient, so in addition to scores and ground-truth labels our loss function also accepts a `mask` array that tells it which elements of the scores count towards the loss.\n",
    "\n",
    "但是有一个问题:因为我们操作的是小批量，不同的标注可能有不同的长度，所以我们在每个标注的后面附加`<NULL>`标记，这样它们的长度都是相同的。\n",
    "\n",
    "我们不希望这些`<NULL>`标记计入损失或梯度，所以除了分数和ground-truth标签外，我们的损失函数还接受一个`mask` 数组，该数组告诉它分数的哪些元素计入损失。\n",
    "\n",
    "Since this is very similar to the softmax loss function you implemented in assignment 1, we have implemented this loss function for you; look at the `temporal_softmax_loss` function in the file `cs231n/rnn_layers.py`.\n",
    "\n",
    "由于这与你在作业1中实现的softmax损失函数非常相似，我们为你实现了这个损失函数；查看文件`cs231n/rnn_layers.py`中的`temporal_softmax_loss`函数。\n",
    "\n",
    "Run the following cell to sanity check the loss and perform numeric gradient checking on the function. \n",
    "\n",
    "运行以下cell来检查损失并对函数执行数值梯度检查。\n",
    "\n",
    "You should see an error for dx on the order of e-7 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Sanity check for temporal softmax loss\n",
    "from cs231n.rnn_layers import temporal_softmax_loss\n",
    "\n",
    "N, T, V = 100, 1, 10\n",
    "\n",
    "def check_loss(N, T, V, p):\n",
    "    x = 0.001 * np.random.randn(N, T, V)\n",
    "    y = np.random.randint(V, size=(N, T))\n",
    "    mask = np.random.rand(N, T) <= p\n",
    "    print(temporal_softmax_loss(x, y, mask)[0])\n",
    "  \n",
    "check_loss(100, 1, 10, 1.0)   # Should be about 2.3\n",
    "check_loss(100, 10, 10, 1.0)  # Should be about 23\n",
    "check_loss(5000, 10, 10, 0.1) # Should be about 2.3\n",
    "\n",
    "# Gradient check for temporal softmax loss\n",
    "N, T, V = 7, 8, 9\n",
    "\n",
    "x = np.random.randn(N, T, V)\n",
    "y = np.random.randint(V, size=(N, T))\n",
    "mask = (np.random.rand(N, T) > 0.5)\n",
    "\n",
    "loss, dx = temporal_softmax_loss(x, y, mask, verbose=False)\n",
    "\n",
    "dx_num = eval_numerical_gradient(lambda x: temporal_softmax_loss(x, y, mask)[0], x, verbose=False)\n",
    "\n",
    "print('dx error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNN for image captioning\n",
    "Now that you have implemented the necessary layers, you can combine them to build an image captioning model. \n",
    "\n",
    "现在您已经实现了必要的层，您可以将它们组合起来构建一个图像标注模型。\n",
    "\n",
    "Open the file `cs231n/classifiers/rnn.py` and look at the `CaptioningRNN` class.\n",
    "\n",
    "打开文件`cs231n/classifiers/rnn.py`然后查看`CaptioningRNN`类。\n",
    "\n",
    "Implement the forward and backward pass of the model in the `loss` function. \n",
    "\n",
    "For now you only need to implement the case where `cell_type='rnn'` for vanialla RNNs; you will implement the LSTM case later. \n",
    "\n",
    "在`loss`函数中实现模型的前向和反向传播。\n",
    "\n",
    "目前，你只需要实现`cell_type='rnn'`情况下的普通RNNs；稍后将实现LSTM用例。\n",
    "\n",
    "After doing so, run the following to check your forward pass using a small test case; \n",
    "\n",
    "这样做之后，用一个小测试用例运行下面的代码来检查你的前向传播;\n",
    "\n",
    "you should see error on the order of `e-10` or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "N, D, W, H = 10, 20, 30, 40\n",
    "word_to_idx = {'<NULL>': 0, 'cat': 2, 'dog': 3}\n",
    "V = len(word_to_idx)\n",
    "T = 13\n",
    "\n",
    "model = CaptioningRNN(word_to_idx,\n",
    "          input_dim=D,\n",
    "          wordvec_dim=W,\n",
    "          hidden_dim=H,\n",
    "          cell_type='rnn',\n",
    "          dtype=np.float64)\n",
    "\n",
    "# Set all model parameters to fixed values\n",
    "for k, v in model.params.items():\n",
    "    model.params[k] = np.linspace(-1.4, 1.3, num=v.size).reshape(*v.shape)\n",
    "\n",
    "features = np.linspace(-1.5, 0.3, num=(N * D)).reshape(N, D)\n",
    "captions = (np.arange(N * T) % V).reshape(N, T)\n",
    "\n",
    "loss, grads = model.loss(features, captions)\n",
    "expected_loss = 9.83235591003\n",
    "\n",
    "print('loss: ', loss)\n",
    "print('expected loss: ', expected_loss)\n",
    "print('difference: ', abs(loss - expected_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to perform numeric gradient checking on the `CaptioningRNN` class; \n",
    "\n",
    "运行以下cell对`CaptioningRNN`类执行数值梯度检查;\n",
    "\n",
    "you should see errors around the order of `e-6` or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(231)\n",
    "\n",
    "batch_size = 2\n",
    "timesteps = 3\n",
    "input_dim = 4\n",
    "wordvec_dim = 5\n",
    "hidden_dim = 6\n",
    "word_to_idx = {'<NULL>': 0, 'cat': 2, 'dog': 3}\n",
    "vocab_size = len(word_to_idx)\n",
    "\n",
    "captions = np.random.randint(vocab_size, size=(batch_size, timesteps))\n",
    "features = np.random.randn(batch_size, input_dim)\n",
    "\n",
    "model = CaptioningRNN(word_to_idx,\n",
    "          input_dim=input_dim,\n",
    "          wordvec_dim=wordvec_dim,\n",
    "          hidden_dim=hidden_dim,\n",
    "          cell_type='rnn',\n",
    "          dtype=np.float64,\n",
    "        )\n",
    "\n",
    "loss, grads = model.loss(features, captions)\n",
    "\n",
    "for param_name in sorted(grads):\n",
    "    f = lambda _: model.loss(features, captions)[0]\n",
    "    param_grad_num = eval_numerical_gradient(f, model.params[param_name], verbose=False, h=1e-6)\n",
    "    e = rel_error(param_grad_num, grads[param_name])\n",
    "    print('%s relative error: %e' % (param_name, e))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overfit small data\n",
    "Similar to the `Solver` class that we used to train image classification models on the previous assignment, on this assignment we use a `CaptioningSolver` class to train image captioning models. \n",
    "\n",
    "类似于我们在之前的作业中用来训练图像分类模型的`Solver`类，在这次作业中我们使用`CaptioningSolver`类来训练图像标注模型。\n",
    "\n",
    "Open the file `cs231n/captioning_solver.py` and read through the `CaptioningSolver` class; it should look very familiar.\n",
    "\n",
    "打开文件`cs231n/captioning_solver.py`然后通读`CaptioningSolver`类；看起来应该很眼熟。\n",
    "\n",
    "Once you have familiarized yourself with the API, run the following to make sure your model overfits a small sample of 100 training examples. \n",
    "\n",
    "一旦您熟悉了API，运行以下代码以确保您的模型能够满足100个训练示例的小样本。\n",
    "\n",
    "You should see a final loss of less than 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(231)\n",
    "\n",
    "small_data = load_coco_data(max_train=50)\n",
    "\n",
    "small_rnn_model = CaptioningRNN(\n",
    "          cell_type='rnn',\n",
    "          word_to_idx=data['word_to_idx'],\n",
    "          input_dim=data['train_features'].shape[1],\n",
    "          hidden_dim=512,\n",
    "          wordvec_dim=256,\n",
    "        )\n",
    "\n",
    "small_rnn_solver = CaptioningSolver(small_rnn_model, small_data,\n",
    "           update_rule='adam',\n",
    "           num_epochs=50,\n",
    "           batch_size=25,\n",
    "           optim_config={\n",
    "             'learning_rate': 5e-3,\n",
    "           },\n",
    "           lr_decay=0.95,\n",
    "           verbose=True, print_every=10,\n",
    "         )\n",
    "\n",
    "small_rnn_solver.train()\n",
    "\n",
    "# Plot the training losses\n",
    "plt.plot(small_rnn_solver.loss_history)\n",
    "plt.xlabel('Iteration')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Training loss history')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test-time sampling\n",
    "Unlike classification models, image captioning models behave very differently at training time and at test time. \n",
    "\n",
    "At training time, we have access to the ground-truth caption, so we feed ground-truth words as input to the RNN at each timestep. \n",
    "\n",
    "At test time, we sample from the distribution over the vocabulary at each timestep, and feed the sample as input to the RNN at the next timestep.\n",
    "\n",
    "与分类模型不同，图像标注模型在训练时和测试时的表现非常不同。\n",
    "\n",
    "在训练时，我们可以访问ground-truth标注，因此我们在每一步都向RNN输入ground-truth单词。\n",
    "\n",
    "在测试时，我们在每个时间步上从词汇表上的分布来抽样，并在下一个时间步上将抽样作为输入提供给RNN。\n",
    "\n",
    "In the file `cs231n/classifiers/rnn.py`, implement the `sample` method for test-time sampling. \n",
    "\n",
    "After doing so, run the following to sample from your overfitted model on both training and validation data. \n",
    "\n",
    "在文件`cs231n/classifiers/rnn.py`中实现用于测试时间抽样的`sample`方法。\n",
    "\n",
    "这样做之后，从您的过拟合模型中对训练和验证数据运行以下示例来抽样。\n",
    "\n",
    "The samples on training data should be very good; \n",
    "\n",
    "the samples on validation data probably won't make sense.\n",
    "\n",
    "训练数据上的样本应该很好；\n",
    "\n",
    "验证数据上的样本可能没有意义。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for split in ['train', 'val']:\n",
    "    minibatch = sample_coco_minibatch(small_data, split=split, batch_size=2)\n",
    "    gt_captions, features, urls = minibatch\n",
    "    gt_captions = decode_captions(gt_captions, data['idx_to_word'])\n",
    "\n",
    "    sample_captions = small_rnn_model.sample(features)\n",
    "    sample_captions = decode_captions(sample_captions, data['idx_to_word'])\n",
    "\n",
    "    for gt_caption, sample_caption, url in zip(gt_captions, sample_captions, urls):\n",
    "        plt.imshow(image_from_url(url))\n",
    "        plt.title('%s\\n%s\\nGT:%s' % (split, sample_caption, gt_caption))\n",
    "        plt.axis('off')\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "# INLINE QUESTION 1\n",
    "\n",
    "In our current image captioning setup, our RNN language model produces a word at every timestep as its output. \n",
    "\n",
    "However, an alternate way to pose the problem is to train the network to operate over _characters_ (e.g. 'a', 'b', etc.) as opposed to words, so that at it every timestep, it receives the previous character as input and tries to predict the next character in the sequence. \n",
    "\n",
    "在当前的图像标注设置中，我们的RNN语言模型在每一步都生成一个单词作为输出。\n",
    "\n",
    "然而，提出这个问题的另一种方法是训练网络对字符(e.g. 'a', 'b', etc.)进行操作，而不是单词，因此，在每一步，它接收前一个字符作为输入，并试图预测序列中的下一个字符。\n",
    "\n",
    "For example, the network might generate a caption like\n",
    "\n",
    "例如，网络可能会生成类似这样的标注\n",
    "\n",
    "'A', ' ', 'c', 'a', 't', ' ', 'o', 'n', ' ', 'a', ' ', 'b', 'e', 'd'\n",
    "\n",
    "Can you describe one advantage of an image-captioning model that uses a character-level RNN? \n",
    "\n",
    "Can you also describe one disadvantage? \n",
    "\n",
    "你能描述一下使用字符级RNN的图像标注模型的一个优点吗?\n",
    "\n",
    "你能描述一个缺点吗?\n",
    "\n",
    "HINT: there are several valid answers, but it might be useful to compare the parameter space of word-level and character-level models.\n",
    "\n",
    "有几个有效的答案，但比较字级和字符级模型的参数空间可能是有用的。\n",
    "\n",
    "**Your Answer:** \n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
